numa: discover NUMA topology from ACPI SRAT/SLIT - #1437
Open
gburd wants to merge 2 commits into
Open
Conversation
OSv had no notion of NUMA: the scheduler and allocator treat memory as flat, which leaves performance on the table on multi-socket bare-metal (the large large multi-socket bare-metal hosts). This is the first, discovery-only step. It adds a numa:: module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance Information Table) at boot, right after acpi::init(), and exposes the topology: - numa::nr_nodes() / numa::available() - numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the APIC ids the MADT parse recorded on each sched::cpu - numa::distance(from, to) -- from SLIT (10 == local per ACPI convention), defaulting to 10 local / 20 remote when no SLIT is present - numa::memory_ranges() -- physical ranges tagged with their node It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity subtables, and only trusts SLIT if its locality count matches the node count SRAT reported. On a machine with no SRAT (the common single-node VM) it reports one flat node and available() == false; nothing changes behavior. This intentionally does NOT yet change allocation or scheduling; it only makes the topology available so a node-aware allocator, scheduler affinity, and mbind/get_mempolicy can build on it. Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
OSv has no notion of NUMA: the scheduler and allocator treat memory as flat,
which leaves performance on the table on multi-socket bare-metal (large
multi-socket bare-metal hosts).
This is the first, discovery-only step of NUMA support. It adds a
numa::module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT
(System Locality Distance Information Table) at boot, right after
acpi::init(), and exposes the topology.API
numa::nr_nodes()/numa::available()numa::node_of_cpu(cpu_id)-- resolved by correlating SRAT APIC ids with theAPIC ids the MADT parse already recorded on each
sched::cpunuma::distance(from, to)-- from SLIT (10 == local, per ACPI convention),defaulting to 10 local / 20 remote when no SLIT is present
numa::memory_ranges()-- physical ranges tagged with their nodeIt handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity
subtables, and only trusts SLIT if its locality count matches the node count
SRAT reported. On a machine with no SRAT (the common single-node VM) it reports
one flat node and
available() == false.Scope
This intentionally does not yet change allocation or scheduling; it only
makes the topology available so later work (a node-aware allocator, scheduler
affinity,
mbind/get_mempolicy) can build on it. Nothing changes behavior onexisting single-node setups.
Testing
tests/tst-numa.ccvalidates the invariants (>= 1 node, every CPU maps inrange, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid
nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and
with a QEMU 2-node
-numaconfig (reports 2 nodes, 3 memory ranges,available).
(Recreated from #1418, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)